Skip to content

Add mitigations for intermittent test failures with CUDA OOM - #2484

Merged
juenglin merged 4 commits into
NVIDIA:mainfrom
juenglin:sync-on-teardown
Aug 4, 2026
Merged

Add mitigations for intermittent test failures with CUDA OOM #2484
juenglin merged 4 commits into
NVIDIA:mainfrom
juenglin:sync-on-teardown

Conversation

@juenglin

@juenglin juenglin commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Addresses #2381.

Test-suite hygiene for the address-space exhaustion described in this comment. Two of the three contributing factors are in the tests themselves, and this PR takes both.

Deterministic release at test boundaries. _init_cuda_context now collects and
drains before dropping the context. A pool's virtual address reservation is not
returned until the pool is destroyed and the stream-ordered frees of its outstanding
allocations retire; previously neither was forced between tests, so reservations from
earlier tests stayed live while later tests created their own.

Right-sized pools. Seven pool constructions passed options without setting
max_size, so each reserved a system-dependent window scaling with device memory while
holding at most 1 KiB:

  • test_memory_peer_access.py — four device pools, including the isolated_dmr_x2
    fixture (once per test across seven tests) and a comprehension that reserved one
    window per device. All hold at most NBYTES = 1024.
  • test_memory.py — three pinned pools in the numa_id tests, which only read
    mr.numa_id and never allocate.
    All now pass the suite-standard POOL_SIZE = 2097152, newly introduced to
    test_memory_peer_access.py with a comment explaining why a peer-access test caps a
    pool it barely uses.

Measured impact

On a 23.9 GiB device, 20 uncapped pools exhaust the process address space, versus
125 pools capped at 4 GiB — an uncapped pool costs roughly 1x device memory in
reservation, a 2 MiB cap costs 2 MiB.

Deliberately not included

Managed pools cannot be right-sized: cuMemPoolCreate documents that
CUmemPoolProps.maxSize must be zero for managed pools, so
ManagedMemoryResourceOptions correctly has no max_size option.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot

copy-pr-bot Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Aug 3, 2026
@juenglin

juenglin commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@github-actions

This comment has been minimized.

@juenglin
juenglin marked this pull request as ready for review August 3, 2026 23:41
@juenglin juenglin self-assigned this Aug 3, 2026
@juenglin juenglin added bug Something isn't working P0 High priority - Must do! labels Aug 3, 2026
@juenglin juenglin added this to the cuda.core 1.2.0 milestone Aug 3, 2026
@Andy-Jost
Andy-Jost self-requested a review August 4, 2026 19:07

@Andy-Jost Andy-Jost left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@juenglin
juenglin enabled auto-merge (squash) August 4, 2026 20:35
@juenglin
juenglin merged commit 4fe801c into NVIDIA:main Aug 4, 2026
107 checks passed
@github-actions

This comment has been minimized.

1 similar comment
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Doc Preview CI
Preview removed because the pull request was closed or merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants